Gustavo Diaz
Postdoctoral Fellow
Department of Political Science
McMaster University
gustavodiaz.org
Specific: Different versions of the same survey question change the answers we get
General: Illustrate bias-variance tradeoff in statistics
Context: List experiments
This shows up in causal inference, machine learning, measurement, and many more!
Have you lied about having COVID symptoms?
Would you bribe a police officer to avoid a traffic ticket?
Have you had sex after drinking alcohol?
Have you been offered goods or favors for your vote?
Do you know anyone with links to a militant organization?
Would you oppose a black family moving next door?
Would you allow Muslim immigrants to become citizens?
They are sensitive questions
We can only learn about them using surveys
But asking about them directly leads to misreporting
This form of measurement error is called sensitivity bias
Honesty appeals
Communicating confidentiality protocols
Randomized response
Network scale-up
Endorsement experiments
List experiments
Honesty appeals
Communicating confidentiality protocols
Randomized response
Network scale-up
Endorsement experiments
List experiments
Here is a list of things that some people have done.
Please listen to them and then tell me HOW MANY of them you have done in the past two years.
Do not tell me which ones. Just tell me HOW MANY:
Do not tell me which ones. Just tell me HOW MANY:
\[ \text{Prevalence(Voted yes)} =\\ \text{Mean(List with sensitive item)} -\\ \text{Mean(List without sensitive item)} \]
Estimate: Proportion of individuals in the target population who hold the sensitive attitude or behavior
But we do not know how individual respondents voted!
Did you vote YES or NO on the Personhood Initiative, which appeared on the November 2011 Mississippi General Election Ballot?
\[ \text{Prevalence(Voted yes)} =\\ \text{Mean(Voted yes)} \]
List A
List B
Organization X (advocating for immigration reduction and measures against undocumented immigration)
Randomly appears in list A or B
Single list: Half of the respondents see sensitive item
Double list: Everyone sees it
Equivalent to making two parallel list experiments
\[ \text{Prevalence}_A = \text{Mean}(A_t) - \text{Mean}(A_c) \]
\[ \text{Prevalence}_B = \text{Mean}(B_t) - \text{Mean}(B_c) \]
\[ \text{Prevalence}_{Pooled} = (\text{Prevalence}_A + \text{Prevalence}_B)/2 \]
Need the two questions to measure the same thing
Advice: Pick baseline lists that are as similar as possible
BUT that makes it easier to spot the sensitive item
Direct question: High bias, high precision
List experiment: Low bias, low precision
Double list experiment: Low bias, high precision
(but easier to break)
Imagine something people tend to hide
How common do you think it is among your friends/peers/community?
Design a list experiment to learn about it
How many people would you ask?
How many lists?
How many and which baseline items?
Would it be any different from asking directly?
Different players different winning strategies
Improving precision without sacrificing unbiasedness
Preferring precision over unbiasedness
Applications: Accountability, governance, and representation in low/medium income countries
1. No liars
Those who do not hold the sensitive item never falsely claim to bear it.
2. No design effects
Including the sensitive item does not change how participants respond to the baseline items
Frowned-upon attitude
Pretend-to attitudes
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| Observed response | \(Y_{i1}\) | \(Y_{i2}\) | \((Y_{i1} - Y_{i2})\) |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| \(z_i = 1\) | 1 | 1 | 0 |
| \(z_i = 0\) | 2 | 1 | 1 |
| Inflation | |||
| \(z_i = 1\) | 3 | 3 | 0 |
| \(z_i = 0\) | 2 | 3 | -1 |
Things done in the last six months:
Things done in the last six months:
Items paired by number:
For this question, I want you to answer yes or no.
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw.
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no.
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes.
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.
[TURN AWAY FROM RESPONDENT]
Now you throw the dice so that I cannot see what comes out.
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.
[TURN AWAY FROM RESPONDENT]
Have you thrown the dice?
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.
[TURN AWAY FROM RESPONDENT]
Have you picked it up?
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.
Now, during the height of the conflict in 2007 and 2008 (in Afghanistan), did you know any militants, like a family member, a friend, or someone you talked to on a regular basis?
Please, before you answer, take note of the number you rolled on the dice.